audit: revalidate cross-framework comparisons - #79

Scope

Revalidate the current comparison suite against VectorBT OSS and Pro, Backtrader, Zipline Reloaded, and LEAN. Correct adapters, profiles, evidence, benchmarks, and publication-facing claims, then retain a clean rerun.

Coordination milestone: audit-2026-08

Current result

• All 12 required real-strategy and framework pairs pass across four real-data workloads: ETF allocation, CME futures, crypto perpetual funding, and USD-quoted FX allocation.
• Eight other workload and framework pairs remain explicitly unsupported and receive no parity credit.
• Every required pair replays the same content-addressed historical market data and frozen model-derived targets in both engines.
• Fill quantities, prices, and other nonmonetary fields compare at 1e-8. Account monetary values use a $0.01 round-half-even quantum. The largest retained raw monetary gap is $0.00000015, or 0.000015 cents.
• Transaction costs and position rules are disabled on both sides. The result tests target sizing, order sequencing, fills, cash and margin accounting, applicable funding, and valuation. It does not reproduce each case study's complete production result.
• Performance evidence contains one isolated warm-up and ten isolated measurements per side for all 12 correctness-passing pairs. Only the engine call is timed.

The separate synthetic convention matrix has 66 required passes and two declared unsupported rows. The corrected 250-asset, 5,040-session stress workload passes for all five framework profiles. These remain secondary diagnostic and scale evidence, not the primary realistic-strategy result.

Material correction

The VectorBT stress adapters previously omitted call_seq, so the native engine used column order while the documented ML4T profile used automatic sequencing. The first divergence occurred at fill 58,324. The corrected adapters pass call_seq="auto"; regression tests cover the setting, and the clean stress rerun passes all five rows.

Retained evidence

• validation/REAL_STRATEGY_RESULTS.json
• validation/REAL_STRATEGY_PERFORMANCE.json
• validation/CORRECTNESS_RESULTS.json
• validation/LARGE_SCALE_RESULTS.json
• validation/METHODOLOGY.md

Publication checks reject stale or incomplete evidence by validating engine and adapter digests, the full required and unsupported matrix, negative controls, comparison surfaces, and performance linkage.

Verification

• Clean correctness rerun from engine commit c4b1a6b3fc4422d50840cde41ffd5d5990f2f17a: 12/12 required pairs pass, 8 unsupported
• Fresh engine-only benchmark: 1 warm-up and 10 measured processes per side for 12 pairs
• uv run ruff check src tests validation: passed
• uv run ruff format --check src tests validation: passed
• uv run ty check: passed
• uv run pytest: 2,124 passed, 14 skipped, 87.14% coverage
• pre-commit run --all-files: passed